Skip to content

ci: test gpu on self-hosted runners #108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Conversation

galargh
Copy link
Contributor

@galargh galargh commented Jun 28, 2025

Related to filecoin-project/rust-fil-proofs#1775

Similar to filecoin-project/rust-fil-proofs#1785

This PR enables the job that requires running on a machine with a GPU. It will run on a g6e.2xlarge runner.

@galargh
Copy link
Contributor Author

galargh commented Jun 28, 2025

I see that not only the job that I'm migrating to self-hosted runners, but also Clippy and Test in release mode on MacOS have started failing now with the same error:

error[E0282]: type annotations needed
   --> src/seal.rs:339:5
    |
339 |     filecoin_proofs_v1::clear_cache(cache_path)
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cannot infer type of the type parameter `Tree` declared on the function `clear_cache`
    |
help: consider specifying the generic argument
    |
339 |     filecoin_proofs_v1::clear_cache::<Tree>(cache_path)
    |                                    ++++++++

error[E0282]: type annotations needed
   --> src/seal.rs:426:5
    |
426 |     filecoin_proofs_v1::clear_synthetic_proofs(cache_path)
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cannot infer type of the type parameter `Tree` declared on the function `clear_synthetic_proofs`
    |
help: consider specifying the generic argument
    |
426 |     filecoin_proofs_v1::clear_synthetic_proofs::<Tree>(cache_path)
    |                                               ++++++++

Last month they were still passing on the default branch: https://github.com/filecoin-project/rust-filecoin-proofs-api/actions/runs/15213629917

@vmx Do you have an idea why that might be?

cc @BigLep

@vmx
Copy link
Contributor

vmx commented Jun 30, 2025

Things should work, but the current master patches the rust-fil-proofs crates to point to the master branch:

[patch.crates-io]
filecoin-proofs = { git = "https://github.com/filecoin-project/rust-fil-proofs" }
fr32 = { git = "https://github.com/filecoin-project/rust-fil-proofs" }
filecoin-hashers = { git = "https://github.com/filecoin-project/rust-fil-proofs" }
storage-proofs-core = { git = "https://github.com/filecoin-project/rust-fil-proofs" }
.

As there was a release of rust-fil-proofs, this repo should be updated to use the released version. So I suggest that the current maintainers update to those versions and the we'll see if things still fail.

@galargh
Copy link
Contributor Author

galargh commented Jul 7, 2025

I added a release commit here - c5246a9 - and the workflow now passes.

Now, the question is, what the release process is that we should follow? I see that the previous tags were created manually - https://github.com/filecoin-project/rust-filecoin-proofs-api/tags

@galargh galargh marked this pull request as ready for review July 7, 2025 13:42
@galargh galargh requested review from Copilot, vmx and BigLep July 7, 2025 13:42
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates the crate to v19.0.0, adds release notes, and configures the CI to run GPU tests on a self-hosted runner.

  • Bump version and dependencies in Cargo.toml to 19.0.0
  • Add a 19.0.0 release section and update link references in CHANGELOG.md
  • Extend CI workflow to target a GPU-equipped self-hosted runner and install CUDA drivers

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
Cargo.toml Bumped package and filecoin-proofs dependencies to 19.0.0
CHANGELOG.md Added 19.0.0 release notes and updated [Unreleased] and version links
.github/workflows/ci.yml Expanded triggers, switched to a self-hosted GPU runner, and added CUDA driver setup
Comments suppressed due to low confidence (3)

.github/workflows/ci.yml:52

  • Verify that the runner labels exactly match your self-hosted runner configuration; a mismatch in the label '2xlarge+gpu' will prevent the job from ever being picked up.
    runs-on: ['self-hosted', 'linux', 'x64', '2xlarge+gpu']

.github/workflows/ci.yml:61

  • [nitpick] Consider using the official NVIDIA CUDA GitHub Action or baking the drivers into the AMI (per your TODO) to reduce setup time and complexity in each workflow run.
          curl -L -o nvidia-driver-local-repo-ubuntu2404-570.148.08_1.0-1_amd64.deb https://us.download.nvidia.com/tesla/570.148.08/nvidia-driver-local-repo-ubuntu2404-570.148.08_1.0-1_amd64.deb

CHANGELOG.md:10

  • [nitpick] It may help consumers if you add a brief migration note or highlight any breaking changes introduced by the bumped dependencies alongside the release header.
## [19.0.0] - 2025-07-07

@BigLep
Copy link
Member

BigLep commented Jul 7, 2025

Thanks for the updates @galargh.

@galargh : Would it maybe make sense to break this into two PRs (one for the release, and one for the CI adjustment)?

@vmx : are there any steps we should follow for making releases (e.g., any cargo release commands like with rust-fil-proofs)?

@BigLep BigLep requested a review from rvagg July 7, 2025 16:56
@BigLep BigLep added this to FilOz Jul 7, 2025
@github-project-automation github-project-automation bot moved this to 📌 Triage in FilOz Jul 7, 2025
@BigLep BigLep moved this from 📌 Triage to ⌨️ In Progress in FilOz Jul 7, 2025
@rvagg
Copy link
Member

rvagg commented Jul 8, 2025

The release commit in here had me confused, but I think we're just trying to do two separate things at once? I don't think I mind as long as it's not squash merged, but it would have been clearer for reviewing if they were separate PRs.

My only question is about the nvidia drivers--are they already installed on the standard GitHub machines but not on the current AMI that we have access to?

@vmx
Copy link
Contributor

vmx commented Jul 8, 2025

I would do the release separately. It's done similarly as for rust-fil-proofs. It's even simpler. It's a matter of cargo release major. It would do the version bump in the Cargo.toml, push that tags etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: ⌨️ In Progress
Development

Successfully merging this pull request may close these issues.

4 participants